HCL Notes/Domino 8.5 Forum (includes Notes Traveler)

Subject: Scheduled agent kicking off replication: Network operation did not complete

Feedback Type: Problem

Product Area: Domino Server

Technical Area: Error Message

Platform: Windows

Release: 8.5.3

Reproducible: Intermittent

My customer has been having a replication failure issue which suddently started approx. 6 months (running Domino 8.5.3 HF57 a year prior to issue). They have a few scheduled agents which replicates a few databases between non-clustered servers from Server A to Server B. Basically, Server A receives updates from Oracle overnight, then the scheduled agent replicates it to Server B. NOTE: We can't simply create a server connection doc to replicate theses specific DB's since the replication must be kicked off after the Oracle update to Notes is completed which can vary in duration, and the scheduled agent has to do a few process before kicking off the replication.

Approx. 50% of the time the replication fails as seen in the errors below such as "Unable to replicate", "Network operation did not complete in a reasonable amount of time; please retry", "The following notes did not replicate (push) from".

Our workaround is to manually issue a server console command to replicate the databases which works fine. However, my customer is becomign very irate about this workaround for over 6 months. LOL. I ran FIXUP -F -J -O -L, COMPACT -C -I, UPDALL -R -C to try to fix corruptions, but errors continue.

See Comment below I added in regards to DB size and deletion stubs.

PLEASE LET ME KNOW IF YOU HAVE ANY IDEAS. I CONTACTED THE NETWORK TEAM WHO STARTED MONITORING SERVERS MORE CLOSELY AND THEY DON'T SEE ANY NETWORK RELATED ISSUES.

My idea is possibly to create a scheduled progam doc which runs fixup prior to the replication, but since one of the DB's is huge with 983,888 docs this might not be viable.

___________________________________________________________________________

ERROR 1: CPU utilization exceeds configured thresholds.

Originating Server: USFLDB05/Servers-US/SONY
Event Severity: Warning (high)
Event Type: Misc
Event Time: 03/01/2013 09:32:26 AM

Lotus Entries
Probable Cause:
1. The server USFLDB05/Servers-US/SONY is experiencing high CPU utilization.
2. Refer to the details section of this document for additional information.
Possible Solution:
1. Check for unnecessary processes running on the server and evaluate all applications, including Domino, that are running on the server.
___________________________________________________________________________

ERROR 2: The following notes did not replicate (push) from USFLDB05/Servers-US/SONY to USFLWEB3/External/Servers-US/SONY of red\REDLINE2.nsf - see details for more information

COMMENT: red\REDLINE2.nsf has 31,416 docs (Notes Peak revealed 113 deletion stubs; Stubs removed via Space Savers option - no checkbox)

Originating Server: USFLDB05/Servers-US/SONY
Event Severity: Warning (high)
Event Type: Replica
Event Time: 03/01/2013 06:33:53 AM

Lotus Entries
Probable Cause:
1. Some notes did not replicate.
2. This may be a normal situation.
3. There might be an access issue preventing some notes from being replicated.
Possible Solution:
1. See details (if available) for diagnostic information.
2. Look for additional error information for this failure in log.nsf. Locate the information via the Domino Administrator by clicking the Replication tab. Then open the Replication Events view.
Corrective Action:
1. Inspect replication events. This action will bring the Domino Administrator forward and open the replication view of 'log.nsf' on USFLWEB3/External/Servers-US/SONY.
2. Open the database red\REDLINE2.nsf on USFLWEB3/External/Servers-US/SONY to inspect the documents.
3. Inspect and manipulate databases on USFLWEB3/External/Servers-US/SONY using tools from the Domino Administrator 'Files' tab.

DB05 Rep Log: 03/01 06:28 AM - 03/01 06:33 AM
Missed scheduled replication with server USFLWEB3/External/Servers-US/SONY at 03/01/2013 12:30:00 AM. Last replication completion time: 03/01/2013 04:02:29 AM.
Unable to store document in USFLWEB3/External/Servers-US/SONY red\REDLINE2.nsf (NoteID = 225374) from REDLINE2.nsf (NoteID = 120726): Network operation did not complete in a reasonable amount of time; please retry
Partially replicated USFLWEB3/External/Servers-US/SONY red\REDLINE2.nsf (due to previously reported error)
___________________________________________________________________________

ERROR 3: Unable to replicate USFLWEB3/External/Servers-US/SONY red\REDLINEC.nsf: Network operation did not complete in a reasonable amount of time; please retry

COMMENT: red\REDLINEC.nsf has 983,888 docs (Notes Peak revealed 0 deletion stubs; Stubs removed via Space Savers option - no checkbox)

Originating Server: USFLDB05/Servers-US/SONY
Event Severity: Warning (high)
Event Type: Replica
Event Time: 03/01/2013 04:33:25 AM

Probable Cause: The secondary event/error describes the reason for the database replication failure.
Possible Solution: See additional error information for this failure in the Domino Administrator. Locate the information by clicking the Replication tab and then open the Replication Events view.

DB05 Rep Log: 03/01 04:27 AM - 03/01 04:33 AM
Missed scheduled replication with server USFLWEB3/External/Servers-US/SONY at 03/01/2013 12:30:00 AM. Last replication completion time: 03/01/2013 04:02:29 AM.
Unable to store document in USFLWEB3/External/Servers-US/SONY red\REDLINEC.nsf (NoteID = 3927442) from REDLINEC.nsf (NoteID = 3927370): Network operation did not complete in a reasonable amount of time; please retry
Unable to replicate USFLWEB3/External/Servers-US/SONY red\REDLINEC.nsf: Network operation did not complete in a reasonable amount of time; please retry
___________________________________________________________________________

ERROR 4: The following notes did not replicate (push) from USFLDB05/Servers-US/SONY to USFLWEB3/External/Servers-US/SONY of red\REDLINEC.nsf - see details for more information

COMMENT: red\REDLINEC.nsf has 983,888 docs (Notes Peak revealed 0 deletion stubs; Stubs removed via Space Savers option - no checkbox)

Originating Server: USFLDB05/Servers-US/SONY
Event Severity: Warning (high)
Event Type: Replica
Event Time: 03/01/2013 04:33:25 AM

Lotus Entries
Probable Cause:
1. Some notes did not replicate.
2. This may be a normal situation.
3. There might be an access issue preventing some notes from being replicated.
Possible Solution:
1. See details (if available) for diagnostic information.
2. Look for additional error information for this failure in log.nsf. Locate the information via the Domino Administrator by clicking the Replication tab. Then open the Replication Events view.
Corrective Action:
1. Inspect replication events. This action will bring the Domino Administrator forward and open the replication view of 'log.nsf' on USFLWEB3/External/Servers-US/SONY.
2. Open the database red\REDLINEC.nsf on USFLWEB3/External/Servers-US/SONY to inspect the documents.
3. Inspect and manipulate databases on USFLWEB3/External/Servers-US/SONY using tools from the Domino Administrator 'Files' tab.

DB05 Rep Log: 03/01 04:27 AM - 03/01 04:33 AM
Missed scheduled replication with server USFLWEB3/External/Servers-US/SONY at 03/01/2013 12:30:00 AM. Last replication completion time: 03/01/2013 04:02:29 AM.
Unable to store document in USFLWEB3/External/Servers-US/SONY red\REDLINEC.nsf (NoteID = 3927442) from REDLINEC.nsf (NoteID = 3927370): Network operation did not complete in a reasonable amount of time; please retry
Unable to replicate USFLWEB3/External/Servers-US/SONY red\REDLINEC.nsf: Network operation did not complete in a reasonable amount of time; please retry

___________________________________________________________________________

ERROR 5: Unable to replicate USFLWEB3/External/Servers-US/SONY red\REDLINEC.nsf: Network operation did not complete in a reasonable amount of time; please retry

COMMENT: red\REDLINEC.nsf has 983,888 docs (Notes Peak revealed 0 deletion stubs; Stubs removed via Space Savers option - no checkbox)

Originating Server: USFLDB05/Servers-US/SONY
Event Severity: Failure
Event Type: Network
Event Time: 03/01/2013 04:33:25 AM

Lotus Entries
Probable Cause:
1. An attempt was made to contact a server and no response was received in a reasonable amount of time. The server is probably busy handling other requests and was unable to respond quickly.
Possible Solution:
1. Check the server load by doing a SHOW TASKS command on the console. Wait for the load to decrease or spread the load by having users access a different server.

DB05 Rep Log: 03/01 04:27 AM - 03/01 04:33 AM
Missed scheduled replication with server USFLWEB3/External/Servers-US/SONY at 03/01/2013 12:30:00 AM. Last replication completion time: 03/01/2013 04:02:29 AM.
Unable to store document in USFLWEB3/External/Servers-US/SONY red\REDLINEC.nsf (NoteID = 3927442) from REDLINEC.nsf (NoteID = 3927370): Network operation did not complete in a reasonable amount of time; please retry
Unable to replicate USFLWEB3/External/Servers-US/SONY red\REDLINEC.nsf: Network operation did not complete in a reasonable amount of time; please retry
___________________________________________________________________________

DB05
Elapsed time: 4 days 13:47:49
Transactions/minute: Last minute: 40; Last hour: 84; Peak: 65383
Peak # of sessions: 15 at 02/25/2013 12:40:58 PM
Transactions: 112,535,859 Max. concurrent: 80

WEB3
Elapsed time: 3 days 03:40:23
Transactions/minute: Last minute: 13; Last hour: 718; Peak: 54651
Peak # of sessions: 64 at 02/27/2013 11:46:18 AM
Transactions: 9,037,088 Max. concurrent: 80

WEB6
Elapsed time: 12 days 01:18:19
Transactions/minute: Last minute: 30; Last hour: 16; Peak: 64872
Peak # of sessions: 35 at 02/19/2013 01:29:27 PM
Transactions: 5,568,872 Max. concurrent: 40

CB06
Elapsed time: 12 days 01:20:00
Transactions/minute: Last minute: 6; Last hour: 51; Peak: 51464
Peak # of sessions: 44 at 02/28/2013 12:40:15 PM
Transactions: 2,475,437 Max. concurrent: 40

Crucial tools for IBM Lotus Notes and Domino administration and development...

Find the "crucial tools you need to succeed" including product descriptions, downloads, demos and testimonials.
Speed up IBM Lotus Notes and Domino administration and development with these crucial software tools.
Better, stronger, faster productivity for administrators and developers.
Download and try the lite (free) version

Feedback number WEBB95DQCR created by ~Keiko Asatoolyflar on 03/01/2013

Status: Closed
Comments: